{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# XGBoost Parameter Tuning for Otto Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "我们以Kaggle 2015年举办的Otto Group Product Classification Challenge竞赛数据为例，进行XGBoost参数调优探索。\n",
    "\n",
    "竞赛官网：https://www.kaggle.com/c/otto-group-product-classification-challenge/data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 第二步：调整树的参数：max_depth & min_child_weight\n",
    "(粗调，参数的步长为2；下一步是在粗调最佳参数周围，将步长降为1，进行精细调整)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "首先 import 必要的模块"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "from xgboost import XGBClassifier\n",
    "import xgboost as xgb\n",
    "\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.model_selection import StratifiedKFold\n",
    "\n",
    "from sklearn.metrics import log_loss\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "train = pd.read_csv(dpath +\"Otto_train.csv\")\n",
    "#train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Variable Identification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "选择该数据集是因为的数据特征单一，我们可以在特征工程方面少做些工作，集中精力放在参数调优上"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Target 分布，看看各类样本分布是否均衡"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "#sns.countplot(train.target);\n",
    "#pyplot.xlabel('target');\n",
    "#pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "每类样本分布不是很均匀，所以交叉验证时也考虑各类样本按比例抽取"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop ids and get labels\n",
    "y_train = train['target']\n",
    "y_train = y_train.map(lambda s: s[6:])\n",
    "y_train = y_train.map(lambda s: int(s)-1)\n",
    "\n",
    "train = train.drop([\"id\", \"target\"], axis=1)\n",
    "X_train = np.array(train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本不均衡，交叉验证是采用StratifiedKFold，在每折采样时各类样本按比例采样"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# prepare cross validation\n",
    "kfold = StratifiedKFold(n_splits=5, shuffle=True, random_state=3)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "第一轮参数调整得到的n_estimators最优值（699），其余参数继续默认值"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "用交叉验证评价模型性能时，用scoring参数定义评价指标。评价指标是越高越好，因此用一些损失函数当评价指标时，需要再加负号，如neg_log_loss，neg_mean_squared_error 详见sklearn文档：http://scikit-learn.org/stable/modules/model_evaluation.html#log-loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'max_depth': [3, 5, 7, 9], 'min_child_weight': [1, 3, 5]}"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#max_depth 建议3-10， min_child_weight=1／sqrt(ratio_rare_event) =5.5\n",
    "max_depth = range(3,10,2)\n",
    "min_child_weight = range(1,6,2)\n",
    "param_test2_1 = dict(max_depth=max_depth, min_child_weight=min_child_weight)\n",
    "param_test2_1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Applications/anaconda/lib/python2.7/site-packages/sklearn/model_selection/_search.py:667: DeprecationWarning: The grid_scores_ attribute was deprecated in version 0.18 in favor of the more elaborate cv_results_ attribute. The grid_scores_ attribute will not be available from 0.20\n",
      "  DeprecationWarning)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "([mean: -1.89738, std: 0.00740, params: {'max_depth': 3, 'min_child_weight': 1},\n",
       "  mean: -1.89750, std: 0.00731, params: {'max_depth': 3, 'min_child_weight': 3},\n",
       "  mean: -1.89795, std: 0.00730, params: {'max_depth': 3, 'min_child_weight': 5},\n",
       "  mean: -1.84625, std: 0.01128, params: {'max_depth': 5, 'min_child_weight': 1},\n",
       "  mean: -1.84747, std: 0.01117, params: {'max_depth': 5, 'min_child_weight': 3},\n",
       "  mean: -1.84842, std: 0.01125, params: {'max_depth': 5, 'min_child_weight': 5},\n",
       "  mean: -1.81426, std: 0.00648, params: {'max_depth': 7, 'min_child_weight': 1},\n",
       "  mean: -1.81820, std: 0.00657, params: {'max_depth': 7, 'min_child_weight': 3},\n",
       "  mean: -1.82191, std: 0.00701, params: {'max_depth': 7, 'min_child_weight': 5},\n",
       "  mean: -1.80225, std: 0.00421, params: {'max_depth': 9, 'min_child_weight': 1},\n",
       "  mean: -1.80935, std: 0.00393, params: {'max_depth': 9, 'min_child_weight': 3},\n",
       "  mean: -1.81214, std: 0.00311, params: {'max_depth': 9, 'min_child_weight': 5}],\n",
       " {'max_depth': 9, 'min_child_weight': 1},\n",
       " -1.8022529801314739)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xgb2_1 = XGBClassifier(\n",
    "        learning_rate =0.1,\n",
    "        n_estimators=699,  #第一轮参数调整得到的n_estimators最优值\n",
    "        max_depth=5,\n",
    "        min_child_weight=1,\n",
    "        gamma=0,\n",
    "        subsample=0.3,\n",
    "        colsample_bytree=0.8,\n",
    "        colsample_bylevel = 0.7,\n",
    "        objective= 'multi:softprob',\n",
    "        seed=3)\n",
    "\n",
    "\n",
    "gsearch2_1 = GridSearchCV(xgb2_1, param_grid = param_test2_1, scoring='neg_log_loss',n_jobs=-1, cv=kfold)\n",
    "gsearch2_1.fit(X_train , y_train)\n",
    "\n",
    "gsearch2_1.grid_scores_, gsearch2_1.best_params_,     gsearch2_1.best_score_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([ 2.23074861,  2.65763454,  2.69463925,  3.30102944,  3.62995882,\n",
       "         3.59766273,  4.28756704,  4.49844666,  4.48730793,  5.3142704 ,\n",
       "         5.55349984,  4.68907838]),\n",
       " 'mean_score_time': array([ 0.16698532,  0.07552595,  0.10664678,  0.10792656,  0.0900722 ,\n",
       "         0.11578994,  0.12307835,  0.12351956,  0.13793001,  0.17913656,\n",
       "         0.16294136,  0.10563679]),\n",
       " 'mean_test_score': array([-1.89738453, -1.89749741, -1.89794523, -1.84625175, -1.84746907,\n",
       "        -1.84842319, -1.81425999, -1.81819681, -1.82191243, -1.80225298,\n",
       "        -1.80935193, -1.81214152]),\n",
       " 'mean_train_score': array([-1.89582817, -1.89601021, -1.89646037, -1.84306682, -1.84496196,\n",
       "        -1.84615697, -1.80789678, -1.81299341, -1.81775794, -1.79172928,\n",
       "        -1.80148929, -1.80581986]),\n",
       " 'param_max_depth': masked_array(data = [3 3 3 5 5 5 7 7 7 9 9 9],\n",
       "              mask = [False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'param_min_child_weight': masked_array(data = [1 3 5 1 3 5 1 3 5 1 3 5],\n",
       "              mask = [False False False False False False False False False False False False],\n",
       "        fill_value = ?),\n",
       " 'params': ({'max_depth': 3, 'min_child_weight': 1},\n",
       "  {'max_depth': 3, 'min_child_weight': 3},\n",
       "  {'max_depth': 3, 'min_child_weight': 5},\n",
       "  {'max_depth': 5, 'min_child_weight': 1},\n",
       "  {'max_depth': 5, 'min_child_weight': 3},\n",
       "  {'max_depth': 5, 'min_child_weight': 5},\n",
       "  {'max_depth': 7, 'min_child_weight': 1},\n",
       "  {'max_depth': 7, 'min_child_weight': 3},\n",
       "  {'max_depth': 7, 'min_child_weight': 5},\n",
       "  {'max_depth': 9, 'min_child_weight': 1},\n",
       "  {'max_depth': 9, 'min_child_weight': 3},\n",
       "  {'max_depth': 9, 'min_child_weight': 5}),\n",
       " 'rank_test_score': array([10, 11, 12,  7,  8,  9,  4,  5,  6,  1,  2,  3], dtype=int32),\n",
       " 'split0_test_score': array([-1.904837  , -1.9048393 , -1.90537758, -1.86813661, -1.86922186,\n",
       "        -1.87028639, -1.80864564, -1.81205344, -1.81534182, -1.7955135 ,\n",
       "        -1.80454465, -1.80950213]),\n",
       " 'split0_train_score': array([-1.90079462, -1.90087334, -1.90126588, -1.86396653, -1.86575001,\n",
       "        -1.86690596, -1.80110119, -1.8061502 , -1.80999227, -1.78507966,\n",
       "        -1.79747592, -1.80407026]),\n",
       " 'split1_test_score': array([-1.89414433, -1.89418724, -1.89449427, -1.84260162, -1.84371216,\n",
       "        -1.84471012, -1.81933539, -1.82307976, -1.82619675, -1.79917828,\n",
       "        -1.80483572, -1.80860676]),\n",
       " 'split1_train_score': array([-1.89538117, -1.89557858, -1.89589882, -1.84090216, -1.84306947,\n",
       "        -1.84431668, -1.81445103, -1.8197674 , -1.82399422, -1.78883   ,\n",
       "        -1.79693318, -1.80241998]),\n",
       " 'split2_test_score': array([-1.90097688, -1.90122587, -1.90157714, -1.84294482, -1.8439429 ,\n",
       "        -1.84512163, -1.81001654, -1.81433944, -1.81827517, -1.80576014,\n",
       "        -1.81140141, -1.81747666]),\n",
       " 'split2_train_score': array([-1.89983001, -1.90005227, -1.9005272 , -1.83932917, -1.84119965,\n",
       "        -1.84252076, -1.80457729, -1.80961677, -1.81502997, -1.79441037,\n",
       "        -1.80353913, -1.81113164]),\n",
       " 'split3_test_score': array([-1.90253915, -1.90250469, -1.90300397, -1.8420231 , -1.84330308,\n",
       "        -1.84418303, -1.80879493, -1.81282132, -1.81611468, -1.80624298,\n",
       "        -1.8142515 , -1.81280824]),\n",
       " 'split3_train_score': array([-1.90115811, -1.90137115, -1.90198618, -1.84171322, -1.84326367,\n",
       "        -1.84447145, -1.80390443, -1.808715  , -1.81286745, -1.79601218,\n",
       "        -1.8060183 , -1.80577938]),\n",
       " 'split4_test_score': array([-1.8844191 , -1.88472388, -1.88526713, -1.83554503, -1.83715799,\n",
       "        -1.83780728, -1.82451212, -1.82869492, -1.8336392 , -1.80457228,\n",
       "        -1.81172845, -1.81231459]),\n",
       " 'split4_train_score': array([-1.88197695, -1.88217572, -1.88262378, -1.82942303, -1.83152702,\n",
       "        -1.83257   , -1.81544997, -1.82071769, -1.82690579, -1.79431418,\n",
       "        -1.80347994, -1.80569803]),\n",
       " 'std_fit_time': array([ 0.4208097 ,  0.19117497,  0.1765739 ,  0.1672316 ,  0.11324032,\n",
       "         0.15489323,  0.13125475,  0.15735095,  0.17904223,  0.13649619,\n",
       "         0.19436801,  0.38385669]),\n",
       " 'std_score_time': array([ 0.03265332,  0.01407311,  0.04169902,  0.04168524,  0.02764364,\n",
       "         0.02179954,  0.0489764 ,  0.03210032,  0.02548046,  0.03983879,\n",
       "         0.0646084 ,  0.05419839]),\n",
       " 'std_test_score': array([ 0.00739684,  0.00730685,  0.00730231,  0.0112763 ,  0.01116656,\n",
       "         0.01125537,  0.00648432,  0.00656567,  0.00701213,  0.00420672,\n",
       "         0.00393359,  0.00311136]),\n",
       " 'std_train_score': array([ 0.00722696,  0.00721537,  0.00723786,  0.01134406,  0.01125399,\n",
       "         0.01126567,  0.00588467,  0.00603469,  0.00654582,  0.00411914,\n",
       "         0.00362054,  0.00292752])}"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gsearch2_1.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best: -1.802253 using {'max_depth': 9, 'min_child_weight': 1}\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYUAAAEFCAYAAAAMk/uQAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJzt3XmcHHWd//FX9TV3kkkyAQTChOuLgCKKhlP4wRI5lWOT\nRQLuAgooiCLKrogLrBziJqLAQjhFAcUVWA6RQ5FDEFwFkfuLBFjjgTlmcszVM91dvz+qpqa6p7un\ne6Z7epJ+Px+PPKa7zk9XZj6fb32r+luO67qIiIgARGodgIiITB0qCiIiElBREBGRgIqCiIgEVBRE\nRCQQq3UAE7Vq1YZx3z7V3t5Md3dfJcOpCMVVHsVVHsVVnk01ro6ONiff9Lo+U4jForUOIS/FVR7F\nVR7FVZ56i6uui4KIiGSraveRMWY+cLm19oCc6ScCXwHWAbdYa28yxkSAa4DdgCTwaWvtm9WMT0RE\nslXtTMEYcy5wI9CYM3028A3gAGB/YLExphM4Cmi01u4F/BuwtFqxiYhIftXsPloOHJNn+rbAH6y1\nXdbaDPBbYE9gX+AhAGvts8AeVYxNRETycKo59pF/BnCHtXbP0LR2vEKwD7ABeBK4FtgLuMta+6C/\n3J+Aba21qWL7SKXS7lS9ECQiMoXlvfto0m9JtdZ2G2POBu4C1gDPA6uB9UBbaNHIWAUBmOgtWaxa\ntWHc61eL4iqP4iqP4irPphpXR0db3umTfveRMSYGfBDYD1gE7AQ87f87zF9mT+ClyY5NRKTeTdqZ\ngjHmeKDVWnu9MQa8M4QBYKm1drUx5n+Ag40xv8Y7rTlpsmITqUeu6+IO/3TB60l2ybiACxm/a9l1\n/Wn409zwuuH1Q9srOG9kWrA9/+ea3iG6u/uy9pt33ZyYXUKvc+f788KfJ5O7/zE+T3NzAz29A9nz\ncuMLTcvebvZ+GRXzGJ+HAvtwYeaMJo7ap5Omhsqm8apeU5gM4/1G8xsr1rL83Q309Q0SPgTefxGQ\nNS33xchy+Q5fydsLvXFDUxsb4/QPDBWIwR01zc2zQbfE+AvF5eZZqaEhzkAyf1z5fo/yxzA6/sIx\nFN/msHgiymAyRZ5ZZfzf5omrwP6KxRWeE49HSQ6mcpJOgQRGkQSbm7AJJbZQws5NhKVsZ6rKpIfY\n8JfnmT53fsnr9K15i2i8iYZpW1QxstKt/b9nSA1sYLZZUOZ6zzJ96w/T3/UO6/70LFt8cHHe5RKx\nCIfs1MdP77kDx4GDDz6URYs+WfJ+Cn2jeaMf5mK87v/1O7zydletw5BNnONAxHH81w6Og//PwSH8\nMzw/zzR/2ai/PcdfIOL/WUf8hRwK7GN4Gg6JRJTUUDp7f6F9EIo52Fd4H3njy95HwRgYjn30Ppqb\nEwwMDAGwfu0qfvbyi3z8k/80xnEZif3eHz3Iru/flx133rHg53Fy9suomEcfsxnTm1m/vr/A58n+\nfwtv97Gf/5X167o4ZtEeWfsvtO7wtNNO+TaXXPQ53rAtPPzgm5x7xj55P89mc1o58vDDuPHGW2lq\nauKEExayYMGhzJgxY0K/s3VbFD77iV3pTWVYt7bfmxCqmc6oF94vDRD8gYQ5eZYrNH9k2sjE3P3N\nbG+hu7t31MrDr8rdXva0PNvL2lD4ZfZnnjmzla6unlH7y7PpovvLF1e+7YVXz56bHdfs2a2sXt2T\nNS0cf/79Fd5e4RhGb8jJWi57A5vKBcr//uWb/Pb1lRWN4cM7zWHRgdsXjOvyy3/EujV/peuPv+Ct\nt95k3bp1AHzxi19hu+2259JLL+LPf15BMplk4cLj2KZzW1Ysf5GerhUcffB8Nt9881H7/NnP7ufp\np58kmUyyZs1qFi78JL/61RO8/fZyzjjjC+y33wHcddePeeKJx+jv72fGjBlceukSHnvsQZ5++lku\nuuhSLr74AnbeeVeOOWZh3s/1hz+8wHe/u4S2tmlEo1F22WVX5m0xjTvvvIOf//xhHMfhoIMWsHDh\ncVxyyYW4rsvKlX+nv7+P88//D1566QXWre3iqiu+wcKFn+Tv7/6Fiy/8Ct3dXeyzz36ccsppPPLI\nQ/T39/HpT/8zt932E2KxGN3dXWQyGeLxiaf0ui0KzY0xtpnCf7TNsQIZqoY62psgNeYNYZOuuTFe\n8X5Vqa1Pfepkli9/k4GBAT70oY9w9NH/yIoVf+LSSy9i6dIreeGF57nuultwHIf//d9n2Wmn9zJ/\n/l4cdNCCvAVhWF9fH1dc8V/84hcP8+Mf/5Drr7+F3//+OX7ykx+xzz4fZd26dXznO9cQiUT40pfO\n5LXXXmHx4sU89tgTXHLJhQwNDRUsCABLl17GxRd/i7lzt2HJkssAePvtt3j00Z9zzTU3AnD22Wcw\nf753l/6WW27F+edfxDPPPMU113yXyy+/gltuuYkLL7yUV155icHBQS67bAmZTIZjjz2cU045jQUL\nDgn2F4vFeOKJX/Ltb1/OXnvtS2Nj04SPvf6SRKSoRQduP6pVP1neeutNnn/+dzz66CMAbNiwnubm\nFs466xy+9a1L6OvrZcGCQ0ve3g47GABaW9vo7JyH4zi0tbWRTA4SiUSIx+NceOHXaGpqYuXKlaT8\nRtDixf/C6aefxE033VZ0+11dXcyduw0A73vfbvz5zyt4663l/P3v7/KFL3zW/wwbWLFiBQAf/OCH\nAdh119248spvj9retttuRyKRACAazZ+u99//QPbb7wAuueRCHnroAQ4//OMlH498NCCeiEw5jhPB\ndTNss00nixYdz9VXX883vvFNFiw4lNWrV2Pta1x22RK+9a3vcO21V5JKpXAcB9fNjLHdwmfgb775\nR5588nH+4z8u4+yzzw22NTg4yJVXLuUrXzmPpUu/ydDQUMFtdHR08M47bwPw2muvAjB37jZ0dm7L\nVVddx9VXX89hhx3BdtvtAIC1rwHw0kt/YN687UKf3fVfF/4sPT09nHnmqQwOegWtqamJSGTiKV1n\nCiIy5bS3tzM0lKKvr4/HHvs59913N319vZx88qnMmjWLrq41nH76yUQiEY477gRisRg777wry5Zd\nzRZbbEln57yy97nVVlvT1NTEZz97MgCzZs1m9epVLFmyhL333pdPfOIYVq9exbJlV/H5z38p7za+\n8pXzuPjiC2hpaaG5uZm2tjZ22GFH9tjjw3zuc6cwODjEe9+7Cx0dHQA8++yveeqpJ8hkMpx33gUA\n7LbbB/jyl8/i5JNPzbuP8DWFgw8+hDPO+AyxWIztttuhrLOmQur2llTYdC4EThbFVR7FVZ56i+uS\nSy7koIMWsOeee49r/Qp8oznveYjOFERkk7JkyTd55523Rk1fuvRKGhoa86xRnnfffZeLL/73UdN3\n3/1DnHLKaRPefq3pTKGOWiYTpbjKo7jKo7jKU60zBV1oFhGRgIqCiIgEVBRERCSgoiAiIgEVBRGZ\ncpLJJPfff09Z67zwwvO8+eYfqxRR+e65505uuum6ste79967SaVSPP/877jggq8WXfbHP76dE05Y\nxJlnnsqZZ57Kn/70zjijHaGiICJTTlfXmrKLwgMP3Mfq1auqFNHkufXW75FOp0ta1trXOf/8i7j6\n6uu5+urrmTu3c8L71/cURKSou9/8Kb9fWdkHIe4+530cs/0RBef/4Ac38847b3PzzdeXNEpqZ+e2\n/OY3z/DGG6/T2bltRUdJvf322yc0SipQ1iipXV1ruPDC81i48JOsWLGCc845q+Aoqda+xm23fY81\na9aw9977cuKJJ034/0ZFQUSmHI2SWtooqQcdtIBjjllES0sL5533ZZ5++lfss89+Ezr2KgoiUtQx\n2x9RtFVfTRoltfAoqa7rsmjR8bS2tgKw11778sc/2gkXBV1TEJEpR6OkljZK6qc+9U/09fXhui7P\nP/9bjNmp6Ocvhc4URGTK0SippY2Seuqpn+Oss04nHo+zxx4fYa+99i37c+fS2Eeb4Jgm1aK4yqO4\nylNvcWmUVBGRSaBRUidGZwp11DKZKMVVHsVVHsVVHo2SKiIiVaeiICIiARUFEREJqCiIiEhARUFE\nphyNkjr2KKmrVq0KRkc988xTOeSQA7jnnjsnEjKgoiAiU5BGSR17lNSOjo5gdNTTTz+THXfciSOP\nPHrC+9f3FESkqFU/uYMNv/ttRbfZtseH6Vh4XMH5GiW1tFFSwRsD6Yor/pMLLvgG0Wh0ov81Kgoi\nMvVolNTSRkkFePrpJ5k3b9uKPEsBVBREZAwdC48r2qqvJo2SWniU1GEPP/wgCyv4/6OiICJTTniU\n1AULdmbBgkPo7u7i/vvvyRolNZlMcuyxh/Oxjx1WsVFSb7jh+wwMDHDKKScAo0dJ/a//uoF4PJ53\nG8OjpHZ2zuO1116lra0tGCV16dIrcRyHH//4drbbbgcef/xRrH2N3Xb7wLhGSR32+uuv8b737Tb2\ngiVSURCRKUejpJY2Smp3dzctLS1Fi125qjr2kTFmPnC5tfaAnOmLgXOANHCztfZaY0wc+D7Q6U//\njLX29bH2obGPJo/iKo/iKk+9xVV3o6QaY84FTgR688xeAuwC9ACvGmPuAD4KxKy1extjDgYuAY6t\nVnwismnSKKkTU7UzBWPMscCLwK3W2j1z5j0MnA6sAX4PfAjYAq8Q/CNwNLDQWjvm1ZNUKu3GYhO/\nDUtEpM5M7pmCtfYuY0xngdkvA8/hnUXcba1da4xpw+s6eh2YDZT0UNju7r5xx1hvp6sTpbjKo7jK\no7jKU4Huo7zTJ/0bzcaY9wOHA/PwisAcY8xC4GzgYWvtjsBuwPeNMRM/1xMRkZLVYpiLdUA/0G+t\nTQMrgXag258H0AXEAfULiYhMokm7JdUYczzQaq293hhzHfCUMWYQWA7cAiSAm40xv/Jfn2etzXeR\nWkREqqSqRcFa+w6wp//6h6Hpy4BlOYsPAouqGY+IbBySySSPPPIgRx55VMnrvPDC87S2trH99jtU\nMbLS3XPPnaxZs6bsO5LuvfduDj/847z44gvce+9dXHTRZQWXfeihB/jRj26lpaWVww47giOOKP14\nFaJRUkVkytEoqWOPktrV1cWNNy7jqquu4+qrr+eRRx7ib3/764T3r280i0hRv/7lct56fWVFt7nt\nTnPY+8DtCs7XKKljj5L6kY/szvbb78C0adMB2GmnnXnllZfYYov3TOj/RkVBRKYcjZI69iipiUSG\nt99+i66uNTQ3t/Dcc79l7ty5Ez72KgoiUtTeB25XtFVfTRoltfAoqdOnT+fzn/8SX/vauUyfPp0d\nd9yJ6dNnlHwsClFREJEpR6Okjj1KaiqV4o03Xueaa25kaGiIs88+g9NOO6Po5y+FioKITDkaJbW0\nUVIBTj55MYlEA8cdt5gZMyZ+plDVUVIng0ZJnTyKqzyKqzz1FlfdjZIqIlILGiV1YnSmUEctk4lS\nXOVRXOVRXOWp1pmCvrwmIiIBFQUREQmoKIiISEBFQUREAioKIiISUFEQEZGAioKIiARUFEREJKCi\nICIiARUFEREJqCiIiEhARUFERAIqCiIiElBREBGRgIqCiIgEVBRERCSgoiAiIgEVBRERCagoiIhI\nQEVBREQCKgoiIhJQURARkYCKgoiIBFQUREQkEKvmxo0x84HLrbUH5ExfDJwDpIGbrbXX+tO/Cnwc\nSADXWGtvqmZ8IiKSrWpFwRhzLnAi0Jtn9hJgF6AHeNUYcwewG7A3sA/QDHy5WrGJiEh+1ew+Wg4c\nU2Dei8B0oBFwABf4GPAS8D/A/cBPqxibiIjk4biuW3QBY8xM4IPW2l/43TsfBC6w1r461saNMZ3A\nHdbaPXOmLwVOwjuLuNta+wVjzA3ANsARwDzgPmAna23RAFOptBuLRccKRUREsjn5JpbSffQj4H5j\nDMBC4ApgGfDR8URhjHk/cDhe4u8BbjPGLATWAK9bawcBa4wZADqAlcW2193dN54wAOjoaGPVqg3j\nXr9aFFd5FFd5FFd5NtW4Ojra8k4vpfuo3Vp7NfAJ4BZr7a14ff7jtQ7oB/qttWm8pN8OPAUcYoxx\njDHvAVrwCoWIiEySUs4UIsaYDwFHAfsbYz5Q4npZjDHHA63W2uuNMdcBTxljBvGuPdxirR00xnwU\n+F+8YnWGXzRERGSSlJLc/xX4T2CJtfYtY8yzwNmlbNxa+w6wp//6h6Hpy/C6oHKXP7eU7YqISHWM\n2X1krX0UONRa+11jzPbAN4Anqh6ZiIhMujGLgjHm68ANxpi5wJPAF4Hrqh2YiIhMvlIuNH8C+Axw\nPHCbtfZgYPeqRiUiIjVRSlGIWmuTeN8f+JkxJoJ3Z5CIiGxiSikKjxpjXsYbj+hJvOsJ91U1KhER\nqYlSLjR/GTgM2NNamwE+b63916pHJiIik66UC80deAPYrTTGrAUuMMZsVvXIRERk0pXSfXQd3hfK\ntgU6gWcADWktIrIJKuXLa9taa8OjnX7LGHNitQISEZHaKeVMwTXGbD38xv++wlD1QhIRkVop5Uzh\n68Azxpjf4A21Oh84rapRiYhITYxZFKy1PzXG7A58BO/M4nRrbdHhrEVEZONU0min1tpVwAPD740x\nL1lr31e1qEREpCbG+zjOzkoGISIiU8N4i0LxZ3iKiMhGabxFQURENkEFrykYYzLkPyNwCkwXEZGN\nXMGiYK3VWYSISAkybgbXdUm7Ge81GdL+tIw/zfvnknHTZBg93SVDOuOtm/G35YaXyVln88GZzI13\nEnEqm6rLftayiGy6XNfFzZOwvKSUZ1qQBF3SbjqUBP3k57oF13NdL3FmcHHdDM3rEqzf0FdgHW9b\nI+uMJGF3jDiLr5PJu79gHTeDE4GhdGp0gg8dp1o57yNns2XrFhXdpoqC1KXhP/C0mw7++IP3meH3\n3ry0/zoTep29ToZ0Jp393k3TvDbO+g39QYJx/eQzkpRCSYZQggonzOEWZNY66TzrZifbDAWSn5sB\nxyWVSZMv2bp11DPs4BBxIkQcB8eJEHUiRIjgOA5RJ4LjRIg4EeLRGDEnRoSIv3z2Og7e+6gTxXGc\n0DIRIuS8D60bwV8/7zpRIjnTHccJrRNhy9mz2SJe+bFJVRQkaB3mS24jCTBNcn0Pq3s2kHZDy2Qy\n2e/d0e8zJSTaUeuUsN3hWCMxGBwcKnG76U0y+TmEE1koqeQkslgkTjwWI5PBT4JOkPyGl8ldZ3hb\nuUkwN9FF/ESXL3HmS7a568yY3kzPhuTIOk7E/1x5km3eOIfXiRCNhJN1JPsz+onacZySjm1HRxur\nVm2o8v9g+aoV15hFwRjz7zmTXKAfeM1a+0CeVTZ6I63I3KRSSiuycPIqtM3h5Dq8nYa/xtjQ2188\n0WbytHDdArFlirRw/e1urCJOhFgkmpUgok7Um+5ESUTjwXsv0Yy8Hp4ecSJEI+FlIjnrjEwbeR8l\nGsm37MgyM6a30LMhOSppRYKkOtIadIZbmqNalrnrjU7IDk7JCQ7qL8lJeUo5U9ge2AH4kf/+WGA9\nsK8xZn9r7bnVCq6a7l3+IL/59XOk0qlNphXp4IxKbuFkFoskgqSVlSAj2clsdAL03rc0NzCYTOdN\nmnkTb852C64TKZJ4cxJt+PMNJ8OpmkymalwixZRSFAzwUf85zRhjlgFPWGv3Msb8Adgoi0I8EqM5\n3ogbdUYnyTFakYWSZjmtyOHt5m4j4kSY1d7K+nUDBRNtVmz+NodPr6tJSU5k01dKUWj3l0v67xNA\nq/96o71t9bB5B/PPHzlmSia5jtltrHKnXlwisukrpShcDfzOGPNTIAocClxljPki8GI1gxMRkck1\nZkvfWnslsAj4K/A28I/W2mvwRk09qbrhiYjIZCrl7iMH2Nf/FwUixpjXrLV/rHZwIiIyuUrpPvoW\n3t1HN+ONe3QSMA/4YhXjEhGRGiilKCwAdrfWZgCMMQ8AL1U1KhERqYlS7h6KkV08YkC6OuGIiEgt\nlXKmcDvwuDFm+Mtrn2Tki2wiIrIJGbMoWGsvNcb8HjgQ78zikk11eAsRkXpX0oB41toHgQeH3xtj\nrrHWfm6s9Ywx84HLrbUH5ExfDJyD1w11s7X22tC8OcBzwMHW2tdLiU9ERCpjvN9IPmGsBYwx5wI3\nAo15Zi8B/gHYBzjHGNPurxMHrsMbcE9ERCbZeItCKUMyLgeOKTDvRWA6XsEIP95zCbAM74tyIiIy\nycb7PIUxhxG11t5ljOksMPtlvC6iXuBua+1aY8y/AKustQ8bY75aaiDt7c3EYtFSFx+lo6Nt3OtW\nk+Iqj+Iqj+IqTz3FVbAoGGMeI3/yd4Cm8e7QGPN+4HC8L8D1ALcZYxYCJwOuMeYfgA8APzDGfNxa\n+26x7XV39403lCk76qfiKo/iKo/iKs+mGlehglLsTOHCce+tuHV41wz6rbVpY8xKoN1a+9HhBYwx\njwOnj1UQRESksgoWBWvtE5XckTHmeKDVWnu9MeY64CljzCDetYdbKrkvEREZn6o+o9la+w6wp//6\nh6Hpy/AuKBda74BqxiUiIvlttA/JERGRylNREBGRgIqCiIgEVBRERCSgoiAiIgEVBRERCagoiIhI\nQEVBREQCKgoiIhJQURARkYCKgoiIBFQUREQkoKIgIiIBFQUREQmoKIiISEBFQUREAioKIiISUFEQ\nEZGAioKIiARUFEREJKCiICIiARUFEREJqCiIiEhARUFERAIqCiIiElBREBGRgIqCiIgEVBRERCSg\noiAiIgEVBRERCagoiIhIIFbrAEREBNxUiszAAJnkgPcz919yADd430/v9FaaDj6cSCJR0ThUFERE\nxsHNZMgM9PtJOklmYAA3ORCalu9fP5lkMngfTvJuKlXW/tdGInR+cD6JLd5T0c+loiAidcHNZMgk\nk37izpOs8yRxNznAqkyKgfU9o1vtg4PjD8ZxiDQ2EmlsItrWRryjg0hjI05joz+9kUhD6HXW9CYi\njY3M2XZL1g1V/gpAVYuCMWY+cLm19oCc6YuBc4A0cLO19lpjTBy4GegEGoCLrbX3VTM+EZm6XNfF\nHRwcu+U9MIAban1nLZ8MJ/jk+INxHCINDV7SbmkmNmuWn6AbcpJ2U1YyL5TknUQCx3EmdHwSM9pg\n1YYJbSOfqhUFY8y5wIlAb57ZS4BdgB7gVWPMHcBRwBpr7YnGmJnAC4CKgshGwnVd3KGhvMnZzWll\n57a6MwMD/DU1SLKnd2T5ZBJcd9zxOMMJu6GR2PQZeVrc4aTdlJPkm4Ll5mw1mzXrB3Ei9XFfTjXP\nFJYDxwC35pn3IjAdSAEO4AI/Ae705zv+vDG1tzcTi0XHHWRHR9u4160mxVUexVWe4bgyQ0Ok+/tD\n/waKvC8yb8B7TyYz7pgiiQTRpkbiTU1E22cQbWoi2uR3sTQ1Be+zf+ab10S0oQEnOv68kGtOU1PF\ntlVJ1fj9qlpRsNbeZYzpLDD7ZeA5vLOIu621a4dnGGPa8IrD+aXsp7u7b9wxdnS0saoKp18TpbjK\nU09xFb5Dpd+72Jlzh0q+LhdnKMlQnzePdHrcsTixWNDSjs6YSXzz3G6U0V0rTm5rPdRqn7P5jLKP\nV8b/NxSe0JuG3vHnhVyb6u9XoYIy6ReajTHvBw4H5uF1H91mjFlorf2JMWZr4H+Aa6y1P5zs2EQq\nLfsOlewLmGP1k4f7wsd7h0qWaJRIQyPxliavO2WzxsL937n95EHybgjeOzHdp7IpqsX/6jqgH+i3\n1qaNMSuBdmPMZsAjwJnW2kdrEJcI4CVydzBJun+ATH//SJIe9XqgyPR+3hwYIFORO1Qaiba2Eps9\nu3CyLnRBMzTdicVxHGfKtnxlapi0omCMOR5otdZeb4y5DnjKGDOId+3hFuA/gXbg68aYr/urHWqt\n7Z+sGGXjNXKnit/S7h/Im6izXw9kT0+OLDteTiJBpKmJSFMziY4OMrF4/pb38J0soy50VvYOFZFy\nOe4Eru5PBatWbRj3B5iqLaZ6iivoIw+1ttPltMj7+2EwSaqvb9wXOZ1YzEvKTaHk3NQUel1oes4y\nDY1ZFzfr6f+xEhRXeSpwTSFvi0OdglI2r5+83BZ5/i6YcfeRRyJBCzsxayaxzTYPJWo/iWe9HmmR\nR0PJ3GlsJBKPV/YAiWzEVBTqRNC9kq+PPJS003kTu7fs236LfCJfAnIavFZ1tKWV2KzZWQk6b4u8\nQGIPd61M1ZacyMZIRWGKywwNjdnaHnldbJmBcX8RyInHiTQ2EW9tJtI2rYQWeYGuloaGuvkCkMjG\nSkWhCtx0ung3Sjh557TI0/39/N9QkqHePq+/fLz3kUejQTKOzZod6jrxukyio5J5/hZ5+NZDtchF\nNn0qCr7RtyHmS+o5Fz2H71YJLzMwgYGy/FsQY83NxKZNIzJnzkjCbvBb203+gFhNjTnTs5cZvv1Q\nRKQcdVsUuh5+kBXPPBW0yCcyzkr4NsRY+8xQq3t0F0rU/1ZnNM8yTiKBE4moRS4iNVO3RSHV3cXQ\n+g04DQ3+sLVj9Y2Hu1kK34YoIrIxq9uiMOe4xezy+dPVIhcRCdGtICIiEqjbMwURkanEdV0yGZfU\nUJrUUIZUKuO9zvszw8xZLWy21TQikcreUKKiICJShJtxGRpKkxpKk05lGBoaSc7pVDp4n/aT9VAq\nTXooQyrlJfeh0LzhaSM/sxN+ufe6LDp5D2bNaa3o51VREJGNUibjJ9pQ63lU0i02bzghD0/Leh9O\n/ON/cFAhsXiEWCxKLB6hoTFGSzw6Mi0WIRYf/um9jsYixOMj86KxCFtuNYPmaYnKx1bxLYpI3XJd\nl3S6hGQdSsDpVJqhVIZELMr69QN+a9xPyP7PfK3tTKayg3lGIk6QmKOxCM2tCeLxKE3NCTKuS8xP\nzNFQso7FRpJ7kMzj4Wmj50WjkYp8h6hat66rKIjUgUzGDbo61kb76F7dO6rrYmgo4yfewv3Ywesi\nLfFKi8ZGkmo8EaWpJT46IYdb1kUScqF50ZiXrPOpt+8NqSiI1Ei5FxZLScjDP9OhRJ9KpcmkK9uq\ndhxCiThKU3OcWLxhdNLNSdbReIS4Py86nOjjEaKxKB0dbWzoGfDfj6yrb+ZPLhUFkQlIpdL09QzS\n1zs46mdIJJlgAAAKTklEQVQm7dLXm6zohcWxRKMO0ZiXaGPxKA1N8VEt5Na2BlLpTHbLOqs1HclK\n+PnmRSJOxZN1R0cbsVW6S77WVBREcriuy2AyRV/PIL1ZiT6Zlfh7ewYZTJb2PIisC4tNcVomkJBz\nf3oXIb3Wdim3J9Zbd4iUR0VB6kYmk6G/d2gkqeck+eB17+CYd5w0NMZoaUswZ4s2mlsSNLcmcn42\nsNXW7axb31exC4sik0FFQTZ6g8kU67r76etJhhJ+uDvHS/79fUNFtxOJODS1JJg5u4Xm1gQtrQma\nWryfzS0NIwm/JUE0NnY3R3NLgt6+8T+QSKQWVBRkSnJdl4H+oQL99cmsbp2hweLPnIgnojS3JJgx\nqzlozbe0NoQSvjetsUnDjYuoKMikSqcz9Pf6ffXhlnwo8ff2DNLfOzjmfehNzXGmTW9kxsxmYn7i\nH93CTxBP6NdcpFT6a5EJc12XocF0kMx7e5KjW/h+8h/oL35hNhJ1aGlJ0LF5qK8+1JofbuE3NceD\n+8p14VSkclQUpKBMZrgLx2vJ/+Xtbt792/qRln7oQu1YX1pKNMRobk0ws6M1q8umubUhq4WfaIip\nC0ekhlQU6lChe+t7e5KhhO+1+ovdR+840NSSYMbM5uzW/PBF2dCF2VhcDyIS2RioKGwihu+tL9RX\nP9K1M/a99TF/3JfNtpzmJ3UvyW+2+TTSbiZo6Tc2Jyo+bK+I1JaKwhQXvre+N3xBNuv+em96eoyh\nDBqbxr63vqU1QTwRzduFo757kU2fikKNDA2ms/rkw7dYpgbTrO3uK+ve+llzWkfdYjmee+tFpL6p\nKFRQoXvrs/rq/Wnl3Fs/cotlw6gWvu6tF5FKUlEoQTqdGfXlqdx76su9tz6488ZP8OEW/txtZrFu\nff8kfToRkRF1WxRc1yU5MET3mr6C99YPt/CrcW99MYmGuv1vEZEaq9vs8/iDltdffLfoMrq3XkTq\nTd0Whc3eM430UIZoPJLdV69760WkjtVtUdj5A+9h/4ONbrEUEQmpalEwxswHLrfWHpAzfTFwDpAG\nbrbWXmuMiQDXALsBSeDT1to3qxmfiIhkq9qN68aYc4EbgcY8s5cA/wDsA5xjjGkHjgIarbV7Af8G\nLK1WbCIikl81v820HDimwLwXgel4BcMBXGBf4CEAa+2zwB5VjE1ERPKoWveRtfYuY0xngdkvA88B\nvcDd1tq1xphpwLrQMmljTMxaW/R+0Pb2ZmKx8V8Q7uhoG/e61aS4yqO4yqO4ylNPcU36hWZjzPuB\nw4F5QA9wmzFmIbAeCH/CyFgFAaC7u2/csUzVsXwUV3kUV3kUV3k21bgKFZRaDIazDugH+q21aWAl\n0A48DRwGYIzZE3ipBrGJiNS1STtTMMYcD7Raa683xlwHPGWMGcS79nALkAIONsb8Gu86w0mTFZuI\niHiqWhSste8Ae/qvfxiavgxYlmeV06sZj4iIFOe4xR6tJSIidUUD7IuISEBFQUREAioKIiISUFEQ\nEZGAioKIiARUFEREJKCiICIigbp5yE6RZzscCfw73jeqb7bW3jBF4job+DSwyp90mrXWTkI8ceBm\noBNoAC621t4Xml+T41VCXDU5Xv6+o8ANgMEb8fd0a+3Lofm1OmZjxVXLYzYHb1DMg621r4em1/rv\nsVBcNTtW/v6fxxsfDuBta+1JoXkVPWZ1URT8ZzuciDcqa3h6HLgC+LA/72ljzH3W2r/XMi7fh4BP\nWWufm4xYQk4A1lhrTzTGzAReAO6Dmh+vgnH5anW8AI4EsNbuY4w5ALgE+ATU/JgVjMtXk2PmH5Pr\n8MZAy51ey7/HvHH5avb7ZYxpBJzchqM/r+LHrF66jwo92+G9wJvW2m5r7SDwFPDRKRAXeL+EXzXG\nPGWM+eokxvQT4Ov+awev9TGslserWFxQu+OFtfYe4FT/7TbA2tDsmh2zMeKC2h2zJXjD3Pw1Z3qt\n/x4LxQU1/P3CexplszHmEWPML/0BQ4dV/JjVRVGw1t4FDOWZlfsMhw14D/+ZFEXiArgDbyyoA4F9\njTFHTFJMPdbaDcaYNuBO4PzQ7JodrzHighodr1B8KWPM94GrgNtDs2r9O1YoLqjBMTPG/Auwylr7\ncJ7ZNTtWY8QFtf396sMrWB/zY7jdGDPcy1PxY1YXRaGI3Gc4tDG6NTXpjDEO8B1r7Wq/+j8A7D6J\n+98aeAy4NTyQITU+XoXiqvXxGmat/WdgR+AGY0yLP7nmv2P54qrhMTsZbzTkx4EPAD8wxmzuz6vl\nsSoY1xT4/XoDuM1a61pr3wDWAFv48yp+zOrimkIRrwE7+H3UPXinXUtqGxLgVf+XjTHvxesnPBDv\nImvVGWM2Ax4BzrTWPpozu2bHa4y4ana8/NhOBLay1l6G16rL+P+gtsesWFw1OWbW2qBrw0/Ap1tr\n3/Un1exYjRFXTX+/8ArW+4DPGWPe48fzN39exY9ZXRaFnGc7fAl4GO+s6WZr7V+mSFzn4bWKk8Cj\n1tqfTVIY5+E99OjrxpjhPvwbgJYaH6+x4qrV8QK4G/ieMeZJIA58ETjaGFPr37Gx4qrlMQvo73FM\nNwG3GGOewruL7GRgUbV+vzR0toiIBOr9moKIiISoKIiISEBFQUREAioKIiISUFEQEZGAioLIJDHG\n3OJ/c3Y8615kjNnPf/24P5aRSMWpKIhsHPYHorUOQjZ9+p6C1B2/lf01vIH1tsMbS2kdcJQ/7TBg\nId4Iti143wL+J7xvjD6Hl6CXA78DvmqtfaDAfhxgKXAE3iBrUeAma+0txphP4X2ZLOJv8wxr7YAx\nZhXwU7wB2DYAi/G+pXoN8C5wNN44Rn/GGwytHfiCtfb+yhwdqXc6U5B6NR84CdgF+CzeYGh7AC8C\nx+EViAOstbsC9wCfs9auAP4VuBa4APh1oYLgOxZvjJxd8IrM9gDGmF2AzwB7W2s/AKwEvuyvMxt4\n3Fr7frxB2K601v4ArwB92lr7kr/cWmvth4Cz8MbSF6kIFQWpVy9ba1dYa/uA1cDweEr/h9f6Ph44\nzhhzGd5zCVoBrLXfwxtv/3jgnDH2cQBwt7V2yFq7ChgeGuH/ATsAzxpjXsB7xsFO/rwB4Af+6+/j\njbOTzz3+z1fwColIRdTl2EciwGDO+/DzGbYGngGuBh7E67bZHYIHnmyN97ezFVDs6Vsu2Q2v4X1E\ngf+21p7lb7OVkb/FjLV2uE83wujnRuRuy8Xr8hKpCJ0piIz2YbwHl1wB/AY4lJGLvN8AfgmcjTfY\nXLG/oV8AC40xDcaYduAQf/rjeAPTzfGvO1yLd30BvIepHOm/PgmvKIFXBNSIk6pTURAZ7REgYox5\nFXgWeAeYZ4zZC+/awNestXcCXYxcCxjFWnsvXgF4Ge/Roa/60/8AXIRXXF7B+zv8ZmjVhcaYF/Ee\nqjJcLB4Clhlj9q7MRxTJT3cfiUwhxhjXWqvuIKkZnY6KTID/hbKrCsw+zFqb73m/IlOWzhRERCSg\nawoiIhJQURARkYCKgoiIBFQUREQkoKIgIiKB/w9Y5xr/zF7EzwAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x116473f10>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# summarize results\n",
    "print(\"Best: %f using %s\" % (gsearch2_1.best_score_, gsearch2_1.best_params_))\n",
    "test_means = gsearch2_1.cv_results_[ 'mean_test_score' ]\n",
    "test_stds = gsearch2_1.cv_results_[ 'std_test_score' ]\n",
    "train_means = gsearch2_1.cv_results_[ 'mean_train_score' ]\n",
    "train_stds = gsearch2_1.cv_results_[ 'std_train_score' ]\n",
    "\n",
    "pd.DataFrame(gsearch2_1.cv_results_).to_csv('my_preds_maxdepth_min_child_weights_1.csv')\n",
    "\n",
    "# plot results\n",
    "test_scores = np.array(test_means).reshape(len(max_depth), len(min_child_weight))\n",
    "train_scores = np.array(train_means).reshape(len(max_depth), len(min_child_weight))\n",
    "\n",
    "for i, value in enumerate(max_depth):\n",
    "    pyplot.plot(min_child_weight, -test_scores[i], label= 'test_max_depth:'   + str(value))\n",
    "#for i, value in enumerate(min_child_weight):\n",
    "#    pyplot.plot(max_depth, train_scores[i], label= 'train_min_child_weight:'   + str(value))\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'max_depth' )                                                                                                      \n",
    "pyplot.ylabel( 'Log Loss' )\n",
    "pyplot.savefig('max_depth_vs_min_child_weght_1.png' )"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 2",
   "language": "python",
   "name": "python2"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
